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ABSTRACT 



Targeted to mental retardation/developmental disability 
(MR/DD) and vocational rehabilitation agencies, developmental disabilities 
councils, consumer and advocate groups, and protection and advocacy programs, 
this manual is designed to assist in the development of state- level data 
systems responsive to current accountability requirements. It also provides 
the basis for improved day and employment services for people with 
disabilities. The manual is organized around three critical issues facing 
MR/DD policy makers and program administrators. The first issue, data 
collection, focuses on critical data sets, data selection, and collection 
criteria, and data collection formats. The second issue, data utilization, 
discusses how data are used for policy development, program development, and 
program monitoring. The third issue, design and implementation, addresses 
potential solutions to barriers identified in a national MR/DD agency 
information system survey. The manual stresses two key themes: (1) an 

increasing need for accountability and responsiveness to contemporary issues 
in MR/DD services related to employment options, community integration and 
participation, choices and decision making, and transitioning to environments 
that are more interdependent, productive, and integrated; and (2) changing 
data system requirements brought about by the significant paradigm shift that 
is occurring in MR/DD services. (Contains 38 references.) (CR) 
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USER'S MANUAL FOR STATE MR/DD INFORMATION SYSTEMS RELATED 



TO DAY AND EMPLOYMENT PROGRAMS 
Overview 

The purpose of this User's Manual is to assist in the development of state-level 
data systems that are responsive to current accountability requirements and that 
provide the basis for improved day and employment services for people with 
disabilities. The Manual's readership includes MR/DD and VR agencies, DD Councils, 
consumer and advocate groups, and Protection and Advocacy programs, as they 
jointly continue their efforts to develop and use Management Information Systems 
(MIS). 

Two significant factors influenced the Manual's development. First, Congress, 
through PL 100-146 (The Developmental Disabilities Assistance and Bill of Rights 
Amendments of 1987), mandated national documentation of services provided to 
people with developmental disabilities. Among other activities, this mandate resulted 
in the Administration on Developmental Disabilities awarding grants to document the 
current status of residential services (Amado, Lakin & Menke, 1990), allocation of 
public resources (Braddock, Hemp, Fujiura, Bachelder & Mitchell 1989), and day- 
employment services (McGaughey, Kiernan, Schalock, Lynch, & Morganstern, 1991). 
Second, as part of the Congressional mandate, we recently completed a national 
survey of state information systems related to day and employment programs (Kiernan, 
McGaughey, Schalock, Lynch & McNally, 1991). The survey results indicated a wide 
variation across state MR/DD agencies with respect to both the type of day and 
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employment data aggregated at the state level and the purposes for which the 
information is used. A summary of these findings is presented in Table 1. 

The Manual is organized around three critical issues facing MR/DD policy 
makers and program administrators. The first issue, data collection, focuses on critical 
data sets, data selection and collection criteria, and data collection formats. The 
second issue, data utilization, relates to how data are used for policy development, 
program development, and program monitoring. The third issue, design and 
implementation, addresses potential solutions to barriers identified in the national 
MR/DD agency information system survey (Kiernan et al., 1991). 

Throughout the Manual two themes occur repeatedly. First, there is an 
increasing need for accountability and responsiveness to contemporary issues in 
MR/DD services related to employment options, community integration and 
participation, choices and decision making, and transitioning to environments that are 
more interdependent, productive, and integrated. Second, data system requirements 
are changing because of the significant paradigm shift that is currently buffeting 
MR/DD services. This shift is best characterized by (Schalock & Kiernan, 1990): 

*A quality revolution, with its emphasis on quality of life, quality 
enhancement mechanisms, and quality assurance. 

‘Living situations and employment in integrated [natural] environments, 
with an emphasis on natural, as opposed to professional, supports. 

‘Personal growth and development, with a focus on decision making, 
empowerment, and choices. 

‘Accountability, with its measurability and reportability requirements 
related to programmatic services, costs, and outcomes. 

At the onset, it is important to clarify what the User's Manual is, and what it is 
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Table 1 

Current Status of State MR/DD Information Systems 
Related to Day and Employment Programs 3 

A. Information System's Structure 

Mainframe in Umbrella Agency (44%) Contract Out (2%) 

Mainframe in State Agency (26%) Personal Computer (28%) 

B. Consumer-Referenced Data Available 

1. Persons served : 49 of the 50 states collect information regarding the 

number of consumers by day/employment service type. In descending 
order, age and primary disability are the consumer characteristics most 
available (42 of 50 states), followed by level of retardation (39 states), 
gender (39), ethnicity (33), and adaptive behavior level (28). 

2. Services provided: 90 percent of the state agencies collect data 
regarding the number of persons placed into supported employment, 
followed by 84 percent for day activity/day habilitation programs, 76 
percent for sheltered employment, and 45% for competitive employment 
or time-limited training. 

3. Costs: Expenditure data are aggregated by funding source in 96% of the 

responding states, and by service category in 95% of the responding 
states. No annualized cost estimates were requested. 

4. Outcomes: 22 states reported the ability to quantify the number and 

services of consumers who moved from one service category to another. 
Sixteen states collect information on the characteristics of consumers 
who moved, 23 states have wage and hour data available, with only 1 1 
states collecting data on benefits received; 46 of the 50 states collect 
unmet needs [waiting list] data; fewer than 13 states (depending on the 
outcome data set) collect information on consumer, employer, or family 
satisfaction, integration with nondisabled coworkers, or other quality of 
work life variables. 

C. Uses of Data 

States are more likely to use the data sets for policy formulation (90%), 
legislative activity (84%), program development (84%), and program 
monitoring (78%) activities. The least reported areas of utilization were 
program evaluation and research (58%), and benefit cost analysis (54%). 

a Adapted from Kieman fit al., (1991). 



not. The Manual is intended to aid state MR/DD agencies to develop a management 
information system (MIS) that can be used for policy development, program 
development, and program monitoring. The critical data sets and procedures 
suggested in the Manual are supported by findings from the national survey of state 
information systems. The Manual is not a guide to computer systems, for each state 
has its own unique needs and information processing systems. 

Users of the Manual can expect a number of benefits from reading and using 
the Manual. First, if the suggestions are implemented, the state's information system 
will be better. Second, at the individual level, users will have a much stronger 
understanding of the: 

‘goals of an information system related to day and employment 
programs; 

‘proper focus of an information system; 

‘design and development standards; 

‘validation procedures; and 
‘data analysis levels and capabilities. 

Alvin Toffler, in his recent book Powershift (Toffler, 1990), suggests that we are 
currently seeing a shift from violence and wealth as sources of power toward a 
knowledge base. Similarly, the current rapid social and political changes necessitate 
a data collection process that is responsive to the rebirth of social activism and 
constituent groups. Both of these trends underscore the importance and availability of 
relevant data for the purposes of policy development, program development, and 
program monitoring. 
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PART I: DATA COLLECTION ISSUES 



Policy makers and program administrators increasingly are asking questions 
about the effectiveness and efficiency of those programs that fall within the domain of 
state MR/DD agencies. Four critical service delivery questions include: 

*Who is being served, and at what cost? 

*What are the outcomes from the services delivered, and how do 
these outcomes compare across groups of persons served? 

*What are the impacts of these services, and do they result in 
people with disabilities moving into more productive and integrated 
environments? 

*Are all persons who request services receiving them? 

This section of the Manual discusses four critical data collection issues related to 
answering these questions and outlines specific techniques to address each issue. 
The four issues include identifying critical data sets, defining each set operationally, 
following data selection and collection criteria, and using standardized data collection 
formats. 

A. Critical Data Sets 

Four critical data sets are suggested to answer the above questions and to 
provide the basis for policy development, program development, and program 
monitoring activities. These data sets, which are summarized in Table 2, include 
numbers of persons served, types of services provided, service costs, and consumer- 
referenced outcomes. 
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Table 2 



Critical MR/DD Data Sets 



Persons Served 

Definitions 

Characteristics 

Services Provided 

Service Categories 
Usage Patterns 



Costs 

Funding Source 
Annualized Costs/Person 

Outcomes 

Movement Patterns 

Current Employment Outcomes (Wages, Hours, Benefits) 
Quality Indicators 
Waiting Lists 



B. Operational Definitions For The Critical Data Sets 



Measurement of the four critical data sets poses significant challenges to state 
agencies due to both conceptual and pragmatic issues. Conceptually, the challenge 
is in reaching consensus that the specific measures outlined in Table 2 actually 
represent the critical data sets; pragmatically, considerable resources are required to 
obtain the information as well as to process and analyze it. This section of the Manual 
addresses the first of these challenges and proposes conceptually sound operational 
definitions for each of the four critical data sets. The definitions proposed are 
consistent with current federal terminology (Administration on Developmental 
Disabilities, 1981; P.L. 95-602) and program practices (Schalock & Kiernan, 1990; 
Schalock & Thornton, 1988). 

1. Persons served . There are two operational definition issues here. One 
relates to the definition used for eligibility determination, and the second relates to the 
characteristics of the persons served. 

A. Definition . The federal adoption of the functional definition of developmental 
disabilities in 1978 provided legislative endorsement of a more individualized 
approach to persons with disabilities by shifting from diagnostic categories to 
assessment of individual functional skills and needs. However, complex issues 
related primarily to the standardized measurement of functional skills within the major 
life activities have resulted in delays by state MR/DD agencies in the adoption and 
implementation of a functional definition. For example, in the most recent survey of 
state MR/DD agencies (McGaughey et al., 1991), only 20 agencies reported that they 
use a functional definition, and for the most part, this information is aggregated only at 
the local service level. Thus, because most state MR/DD agencies do not utilize the 
functional definition to determine service eligibility or to document consumer 
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characteristics, we would propose an operational definition of persons served that 
could include categories such as mental retardation, sensory/neurological, physical, 
and psychiatric. These categories are defined more specifically in Table 3. These 
categories are consistent with those proposed by the Administration on 
Developmental Disabilities (1981), as well as current research in service delivery 
practices (Schalock & Keith, 1988; Schalock & Kiernan, 1990). Currently, 88% of all 
individuals served by state MR/DD agencies have a primary disability of mental 
retardation; of the remaining 12% , 51% have a sensory disability, followed by 
psychiatric (38%) and physical (11%) disabilities (McGaughey et al., 1991). 

B. Char acteristics . The second definitional issue relates to the characteristics 
of persons served. The six demographic characteristics that we propose include age, 
gender, ethnicity, primary disability category, level of retardation (if applicable), and 
adaptive/functional skills (see Table 3). 

2. Services provided . Again, there are two issues: one is defining the 
service categories; the second is documenting the amount or duration of services 
received. 

A. Service categories . The term "service category" is used intentionally to 
focus one's attention on the larger service environment, rather than on specific training 
or treatment techniques used within those environments. The day and employment 
services definitions summarized in Table 4 represent those used in the recent state 
agency MIS survey (Kiernan et al., 1991). These categories represent services funded 
currently at federal and/or state levels and can be used to evaluate the integrated and 
facility based day or employment opportunities for people with disabilities. 

B. Docu menting services . The documentation of services is important for both 
accountability and evaluation purposes. The problem is that documentation is both 
time consuming and expensive. Documentation options vary from intensity to duration 

8 
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Psychiatric: includes conditions such as 
schizophrenic disorders, paranoids disorders, 
major affective disorders, and dual diagnosis. 



Table 4 



Community Day Program/Employment Service Categories 

Time Limited Training For Competitive Employment 

Environment where most workers do not have disabilities 
Time limited job-related supports are provided to the worker with a 
disability in order to maintain employment 

Supported Employment (With Ongoing Support) 

Environment where most workers do not have disabilities 
Ongoing job-related supports are provided to the worker with a 
disability in order to maintain employment 

Sheltered Employment/Work Activity 

Environment where all workers have disabilities 
Continuous job-related supports and supervision provided to all with 
disabilities 

Day Activity/Day Habilitation 

Environment where all participants have disabilities 
Primary program focus: psycho/social skills, activities of daily living, and 
recreation, and/or professional therapies (e.g., O.T., P.T., Speech) 
Continuous supports and supervision are provided to all participants 
with disabilities 

Integrated Day Programs 

Participants are adults or youth transitioning from school (no other age 
restrictions) 

Primary program focus: community integration experiences with 

individuals who do not have disabilities (leisure activities, learning 
activities, etc.) 

Program established to provide an alternative to segregated day 
programs 

Programs For Elderly Individuals 

Environment where all participants are 55 years or older 
Primary program focus: leisure recreation, nonvocational 
May be integrated with elders who do not have disabilities 



measures (Schalock & Thornton, 1988). Most service providers and funding/ 
monitoring agencies generally use a monthly or yearly duration measure. Therefore, 
we propose that the documentation be in months of service, which will then permit one 
to determine costs per participant such as described in the following section. 

3. Costs . (al The two issues here relate to funding source and annualized 
costs per participant. 

A. Funding source . The sources of funding within state MR/DD agencies are 
varied. All but five states in the recent MIS survey reported the availability of 
expenditure, data related to funding source, and all but seven states reported the 
availability of expenditure data by service category. 

B. Annualized costs per person . Being able to account for the cost of a 
program is a critical program management function. Some measure of programmatic 
costs is essential for describing the intensity of services provided, for budgeting 
program replications, and for evaluating whether the impacts produced by the program 
are sufficiently large to justify the program. 

There are a number of different approaches to cost analysis -- all of which are 
beyond the scope of this Manual. The interested reader is referred to Killaugh and 
Leininger (1987) and Moriarity and Allen (1987). However, there are several 
guidelines regarding the operationalization and analysis of program costs, including 
(Schalock & Thornton, 1988): 

(1) Describe the program by service category (see Table 4) and allocate 

all funds used by the program. 

(2) Define the period over which costs will be measured, remembering 

that the cost analysis period should correspond to any related 
impact or benefit/cost analysis time periods (most preferably a 
budgetary period). 



(3) Determine average costs of the program. 

The advantages of using average costs (guideline #3) include their 
comparability with estimates of average program effects, as well as the ability to focus 
on issues of service intensity. It seems reasonable at this point, based on state MR/DD 
agencies' ability to aggregate fiscal data, to propose the following operational 
definition of annualized costs per person (Schalock & Thornton, 1988): 

Average Cost = Total cost for time period 

Per Participant Number of people enrolled 

during that period 

4. Outcomes . The current era of accountability requires that the outcomes 
from MR/DD services be evaluated to determine whether: 

‘public policy goals are being reached: 

‘employment outcomes are improved in integrated employment settings; 

‘people with disabilities experience a high quality of work life; 

‘integrated employment and day services result in increased consumer 
satisfaction: 

‘persons are moving to more productive, integrated environments. 

The four outcome measures listed in Table 5 can provide the basis for 
evaluating the outcomes from MR/DD employment-related programs. These include 
[consumer] movement patterns, current employment outcomes, quality indicators, and 
waiting list information. Operational definitions of each are also given in Table 5. 

In summary, this section of the Manual has suggested operational definitions for 
the four proposed critical data sets related to the people served by state MR/DD 
agencies, the services provided, the costs of those services, and service outcomes. 
Although these critical data sets are desirable, it is generally not feasible to collect (or 
analyze) all the information that people would like. Indeed, the most frequent 
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Proposed Outcome Measures and Their Operational Definitions 
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constraint on data collection or analysis reported by 19 state MR/DD state agencies in 
the 1991 MIS survey was lack of resources (money or persons). Therefore, the 
following section of the Manual discusses a number of data selection and collection 
criteria that will assist users in selecting specific measures within each of the four 
critical data sets. 

C. Data Selection and Collection Criteria 

The issues of data selection and collection are quite different. Data selection 
involves determining what to measure, as reflected in the critical data sets summarized 
in Table 2 (page 6) and defined operationally in the previous section. Data collection 
involves assessing alternative data sources to determine the quality, availability and 
cost of the data. These two concepts are discussed in this section and summarized in 
Table 6. 

1. Data selection criteria . What one measures depends upon the 
program's goals and objectives and the comparisons that one wants to make. In 
general, the following five selection criteria are helpful in determining what to 
measure: 

Attributed to program: an obvious link between service and 
outcomes, 

Sensitive to change and intervention: services affect potential 
outcomes, 

Obtainable: those measures that one has time and tools to actually 
measure, 

Objective: measures that can be easily quantified and thereby 
measured, 

Prioritized: measures that reflect the program's goals and 
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objectives. 



2. Data collection criteria . Once the specific desired data elements have 
been selected from among the four critical data sets, six criteria can be used to assess 
alternative data sources and alternatives. These six include: 

accessibility: available from the primary source (e.g., service provider, 
funding agency), 

completeness: availability of the data element for all service recipients, 
accuracy: reflects actual services or outcomes, 
timeliness: whether data covers the period(s) of interest and how soon 
after such period(s) the data can be collected, 
flexibility: degree to which the data element/set will be influenced by 
shifts in budgets, policies, research needs and/or quality of the 
data source, 

cost: time and resources required to collect and analyze the data 
element/set, 

In addition to these data selection and collection criteria, one also needs to 
consider whether the data will be used for policy development, program development, 
and/or program monitoring purposes. A discussion of these three purposes 
constitutes Part II of the Manual. However, one needs to keep the three uses in mind 
when selecting data sets and committing the resources necessary for data collection. 
What Part II confirms is that the data sets required for policy/program development and 
program monitoring are the same four critical data sets summarized in Table 2 (page 
6) and discussed in the previous section. Users of the Manual will undoubtedly be 
collecting one or more elements from each data set, and therefore the next section of 
the Manual discusses specific data collection formats that can be used for both data 
collection and analysis purposes. 



D. Data Collection Formats 

Collecting information on the desired critical data sets requires resources. 
Therefore, the following three guidelines should be followed: 

‘Data sets chosen should meet the selection and collection criteria 
summarized in Table 6, and should be used for the purposes of policy 
development, program development, and/or program monitoring. 

‘Data sets collected should be clearly defined and easily measured. 

‘Individual data sets should be collected at the provider level and 
aggregated at the state (i.e., centralized or distributed data entry) level. 

Another way to state these three guidelines is to "begin with the end in mind." 
For example, although the majority of states currently aggregate data at the state level, 
many do so only at the regional or county level. This situation makes it very difficult, if 
not impossible, for states to use data meaningfully for policy/program development 
and program monitoring. This section of the Manual outlines a procedure whereby a 
common data input sheet is used by the provider to be shared with the state MR/DD 
agency for state-level, cross-tab and statistical analyses. The authors realize that the 
data input sheet is only a model and may need to be modified to reflect the unique 
needs of each state MR/DD agency. 

1. Data input sheet . A model of the proposed data input sheet that would 
be submitted to the state MR/DD agency by the service provider is shown in Figure 1. 
Note the following aspects of this model format. First, it allows for consumer 
referenced input regarding each of the four critical data sets (persons served, services 
provided, costs, and outcomes). Second, the spread sheet format of Figure 1 
represents a reasonable, inexpensive format for handling the data. Most providers 
have personal computer systems that handle spread sheet/data-based systems very 
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well. Third, a modification of Figure 1 has been used successfully in a number of 
national employment surveys which have found that most service providers are both 
able and willing to complete the data sets (Kiernan, McGaughey & Schalock, 1990; 
Kiernan, McGaughey, Schalock & Rowland, 1988; Schalock, 1988). And finally, the 
proposed input data sheet is not intended either to dictate data collection/analysis 
formats or replace comparable input sheets that are being used successfully by state 
MR/DD agencies. However, it should provide a useful model for states that are either 
developing or modifying their current data collection efforts. 

2. State-level formats . Once the data are available to the state MR/DD 
agency, then the issue becomes how to format it for meaningful analysis and use. A 
number of model formats are presented on subsequent pages; these reflect different 
ways the data can be aggregated to permit policy development, program 
development, and program monitoring. 

A. Service category bv funding source . The data collection format presented in 
Figure 2 provides descriptive information regarding the funding basis for each of the 
service categories. For example, the 1991 MR/DD survey (McGaughey et al., 1991) 
found that 80% of all state resources were allocated to support segregated day and 
employment programs. Similarly, 95% of the remaining resources (comprised 
primarily of federal monies) supported segregated employment programs. 

B. Service category bv primary disability group . The format presented in Figure 
3 can be used to summarize the percent of individuals within each primary disability 
group in each service category and the percent of disability groups in either integrated 
or segregated employment settings. For example, the 1991 survey (McGaughey et al., 
1991) found that integrated employment currently is used less frequently by persons 
with more severe disabilities, and that 75% of all persons in integrated employment 
has a mild or moderate level of mental retardation. Individuals with a primary 
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disability other than mental retardation were somewhat more likely to be in integrated 
environments (14% of those in integrated settings), compared with segregated settings 
(11%). Across integrated employment categories, the largest percentage (19%) of 
people with a disability other than mental retardation was served in time-limited 
training. 

C. Outcomes bv service category . The format presented in Figure 4 can be 
used with individual consumers or aggregated to summarize the current employment 
outcomes and one quality indicator (level of integration with nondisabled coworkers) 
of day/employment services. The format does not contain other quality indicators, 
such as satisfaction measures. However, other quality indicators can be added (as 10 
states indicated they plan to do). Figure 4 also allows for cross analyses with data 
available from the provider-referenced data input sheet (Figure I) to permit more 
sophisticated data analyses such as outcomes by age, disability level, and facility size. 

D. Movement patterns . The format presented in Figure 5 can be used to 
answer important policy and program monitoring questions related to the movement of 
persons into integrated employment environments. For example, our earlier surveys 
(Kiernan et al., 1986; 1988) indicated that there had been movement over the last five 
years from sheltered employment and work activity into more integrated employment 
environments. However, the data reflecting this trend were obtained from individual 
vocational and employment service providers. Thus, the 1991 survey contained 
questions related to the state MR/DD agency's capability to provide consumer 
movement data. In general, few state MR/DD agencies have this capacity. 
Specifically, only 22 states have state-level data related to the number and services of 
consumers who moved during a specific fiscal period, and only 16 have data on the 
characteristics of the consumers who moved. 
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Service Outcomes By Service Category 
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E. Additional state-level formats . The basic matrix format presented in Figures 
2-5 can also be used for the other critical data sets shown on the provider-referenced 
data input sheet (Figure 1). Specifically, costs can be aggregated according to 
consumer characteristics, service categories, primary disability groups, and movement 
patterns. Similarly, waiting list data can be aggregated by the first three variables just 
mentioned. The number of options is limited only by those three guidelines mentioned 
at the beginning of this section: (1) data sets chosen should meet the collection and 
selection criteria summarized in Table 6 and be used for the purposes of policy 
development, program development, and/or program monitoring; (2) the data sets 
collected should be clearly defined and easily collected; and (3) individual data sets 
should be collected at the provider level and aggregated at the state level. 

In summary, Part I of the Manual has suggested four critical data sets (persons 
served, services provided, costs, and outcomes) and their operational definitions that 
can provide the basis for policy development, program development, and program 
monitoring. Data selection and collection criteria were discussed, along with 
examples of a provider-referenced data input sheet and state-level data aggregation 
formats. However, continuing with the notion that one should "begin with the end in 
mind", the Manual's user will need to relate data collection (Part I) to data utilization 
(Part II). It is to the three data utilization issues of policy development, program 
development, and program monitoring that we now turn. 
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PART II: DATA UTILIZATION ISSUES 



Policy makers, program managers, and service providers need to have a clear 
understanding of how data can be used for the purposes of policy development, 
program development, and program monitoring. Definitions of each of these uses 
include: 

Policy Development: Using data to change current policies, rules, 
and/or regulations or to propose new policy initiatives. 

Program Development: Using data either to change existing program 
services or to add new service components. 

Program Monitoring: Using data to determine whether programs are 
meeting their goals and objectives and whether they are in 
compliance with current rules and regulations. 

This section of the Manual outlines the necessary data sets for each of these 
issues and the specific activities that can be undertaken to maximize the use of the 
critical data sets discussed in Part I. A summary of this process is shown in Table 7. 

A. Policy Development 

Recent federal and state policy initiatives have focused on the development of 
supported employment and supported living in order to enhance the a person's 
independence/interdependence, productivity, and community integration of people 
with disabilities. These policy initiatives also reflect the current paradigm shift towards 
quality of life, natural environments, and personal empowerment. Evaluating current 
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Summary of Data Utilization: issues, Necessary Data Sets and Specific Activities 
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service practices and policies and developing new ones require data sets which 
summarize current funding patterns, costs and service outcomes, and activities related 
to outcome and impact analysis. Each is described more fully below. 

1. Data sets. Policy development frequently is based on data 
showing both the feasibility and attractiveness of a particular course of action. For 
example, one of the reasons supported employment emerged so quickly as a national 
and local initiative was its documentation of positive outcomes for persons with severe 
disabilities (Wehman, 1988). However, policy initiatives also need to be evaluated in 
light of their described or anticipated outcomes or impacts. The following data sets are 
suggested for policy development use: 

‘funding patterns 
‘consumer characteristics 
‘costs 
‘outcomes 

2. Activities . The two major data utilization activities related to policy 
development include outcome analysis and impact analysis. The difference between 
these two types of analysis are diagrammed in Figure 6 (Schalock & Thornton, 1988). 

A. Outcome analysis. This activity can be undertaken to answer questions 
regarding the outcomes from habilitation services and whether these outcomes are 
consistent with current or needed policies. As summarized previously in Table 5, the 
suggested outcomes include: 

‘movement patterns 

‘employment outcomes 

‘quality indicators of day/employment services 

‘waiting lists 

B. Impact analysis. Impact analysis focuses on a program's effects or impacts 
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Figure 6. 

Distinction Between Outcomes and impacts 
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on the targeted population. Although an essential part of policy development, impact 
analysis can be quite difficult to accomplish because, as shown in Figure 6, it involves 
making comparisons between what happens in a state program and what happens in 
the comparison state program (e.g., waiver vs. nonwaiver state). Impact analysis 
involves data collection, following people over time, and determining what actually 
happens to the participants, including what would have happened had they not 
received services. Impact analysis addresses a number of policy-relevant questions 
such as: did the program have the intended effects on outcomes, how big are the 
effects, and can these effects be attributed with reasonable certainty to the services 
provided? Thus, impact analysis not only addresses important policy-related issues, 
but also provides the basis for benefit-cost analysis. The interested reader can find 
examples of an impact analysis in Thornton and Fuller (1989) and benefit cost 
analyses in Berkowitz (1988), Conley and Noble (1990), Noble and Conley (1987), 
Schalock and Thornton (1988), and Thornton and Fuller (1989). 

B. Program Development 

Habilitation programs are developed for a number of reasons related to 
policies, unmet needs, political pressure, state and federal policies, and anticipated 
outcomes. The necessary data sets and activities involved in this type of data 
utilization are quite different from those related to policy development. Here, the focus 
is more on describing and evaluating current programs, with the intent being to 
develop new programs by replicating successful programs or modifying existing 
programs through program conversion activities. 

1. Data sets . All four critical data sets, (persons served, services provided, 
costs and outcomes) including unmet need data (waiting lists), are necessary for 
program development efforts. These data sets provide the basis for two activities: 



program replication and program conversion. 



2. Activities . Developing day and employment programs during the I990's 
will probably not involve the creation of "new" programs in the traditional sense. 
Rather, many of these "new" programs will result from replicating currently successful 
employment programs or from converting resources within current programs to provide 
more integrated employment and supports. These activities are discussed in 
subsequent sections. 

& Replication . Replicating a successful program or demonstration project is 

difficult, unless one knows all the details of the program to be replicated. Thus, 
program replication activities need to be based on a thorough process analysis that 
describes the model program's internal and external environments (Schalock & 
Thornton, 1988). The internal environment includes the persons served, selection 
procedures, services provided, costs, and outcomes. The external environment 
includes the organizational structure, organizational philosophy, governance structure, 
resources, funding sources and their certainty, the community, formal/informal linkages 
to other service providers, and the program's constituents. 

Knowledge of each of these internal and external factors is essential for 
successful replication efforts. For example, the external environment of a rural 
program is very different from that of an urban program; similarly, the internal 
environment of a university-based demonstration project is quite different from a 
community-based program's supported employment component. The knowledge 
gained from a good process analysis will significantly improve the successful 
implementation of a program because of the understanding that one has gained about 
how services are provided within the context of the program's organizational structure 
and environment. 

B. Conversion . We have seen recently a strong movement towards systems 
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change, or program conversion, in which sheltered workshop and day activity 
programs devote more of their resources toward integrated employment environments. 
For example, national surveys (Kiernan & Ciborowski, 1985; Kiernan et al. , 1988) of 
facilities providing vocational services to adults with developmental disabilities placed 
17 to 19% of their clientele into nonsheltered settings during the two survey periods of 
1984 to 1986. Although there are a number of conversion models, the conversion 
process typically involves four activities: gaining participant support, determining 
operational procedures, identifying funding sources, and developing staff 
competencies (Bellamy, Rhodes, Mank & Albin, 1988; Kregel, Shafer, Wehman & 
West, 1988 ; Parent & Hill, (1990). The goal of program conversion is not only to 
convert resources to integrated employment, but also for the program to incorporate 
the philosophy and structure of successful integrated employment models. This goal 
is reflected in a number of "best practices", as summarized in Table 8. 

In summary, the large number of persons currently waiting for either new or 
different services is forcing the current state MR/DD systems not only to replicate 
model programs and convert current programs, but also to pursue alternative service 
delivery models such as assessing and using natural supports. Although large-scale 
reliance on natural supports is very new, key components in the development of these 
supports include an ecological inventory of natural supports, a discrepancy analysis 
between the person's needed and available natural supports, assessing needed 
supports (at the individual, co-worker/family/mentor, technological, or service system 
level), and evaluating the outcomes of support against (Schalock & Kiernan, 1991): 
Natural support standards that include: (1) occur in regular, integrated 
environments; (2) performed primarily by people normally working, living, 
or recreating within that environment; (3) individualized and person- 
referenced; (4) coordinated through a person such as a support 
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Table 8 



Best Practices of Agencies With Successful 
Supported Employment Programs 3 

1. Strong leadership, organizational development and philosophical 

commitment to support employment. 

2. High levels of client family involvement. 

3. Provision of a variety of jobs that reflect the range of opportunities available 

in the community. 

4. Use of individual placement, which permits better social integration and job 
matches. 

5. Commitment by disabled employees to the supported employment concept 

and the specific job held. 

6. Assessment techniques that include job analysis and job matching 

7. Use of supports such as assistance from co-workers, provision of transport 

and reasonable job accommodation. 

8. Effective on-site job training of employment specialists. 

9. Business advisory councils or boards including local business persons who 

assist with marketing. 

10. Non-intrusive job coaching and job support that is phased out as soon as is 

practical. 

1 1 . Good connections with university experts and other consultants. 

12. Collaborative efforts among professionals, employers, school personnel, 

state agency staff, persons with disabilities and their families. 

a. Adapted from National Association of Rehabilitation Facilities (1989) . 
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manager; and (5) outcomes evaluated against quality indicators and 
person referenced outcomes. 

Effectiveness criteria that include: (I) less or different external staff 
involvement; (2) attainment of greater levels of integration and stronger 
employment outcomes; (3) increased satisfaction of consumer; and (4) 
demonstrated acceptance and satisfaction of support provider. 

C. Program Monitoring 

Users of this Manual are undoubtedly familiar with a number of different 
approaches to program monitoring including accreditation, compliance reviews, 
certification, and quality assurance evaluations. These are not the types of monitoring 
activities proposed here; rather, the focus of program monitoring should be on the data 
selection, collection, and analysis criteria discussed throughout the Manual. By way of 
review: 

1. Data selection. Four critical data sets have been proposed, including 
persons served, services provided, costs, and outcomes. Specific data elements 
within each set should be selected and monitored based on the selection criteria of: 
attributable to the program, sensitivity to change and intervention, obtainable, and 
objectively defined. 

2. Data collection. Monitoring here should focus on the completeness, 
accuracy and timeliness of the data. 

3 . Data analysis. Monitoring data analysis should focus not only on the 
accuracy and appropriateness of the analyses, but also whether the analyses focus on 
four critical questions asked by policy makers and program administrators: 

‘Who is being served and at what cost? 

*What are the outcomes from the services delivered, and how do these 
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outcomes compare across groups of people served? 

*What are the impacts of these services and do they result in people with 
disabilities moving into more productive and integrated environments? 

‘Are all people requesting services being served? 

There are a number of optional mechanisms for monitoring the above questions. 
Typical activities include systems review at the provider level (Schalock & Kiernan, 
1990) or validation samples of data input sheets, computer [data] entries, and analysis 
and interpretation activities (Schalock & Thornton, 1988). 

In summary, this second part of the Manual has discussed three data utilization 
issues including policy development, program development, and program monitoring. 
Part II of the Manual built upon the concepts and data collection issues discussed in 
Part I. Part III focuses on a number of information system design and implementation 
issues that need to be discussed in order to facilitate and maximize use of the Manual. 
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PART III: INFORMATION SYSTEM 
DESIGN AND IMPLEMENTATION ISSUES 



The purpose of the final section of this Manual is to address the issue of 
information systems and to outline an approach readers might take to increase the 
availability and utility of their state's day and employment service information system. 
The discussion is built around the need for a systematic approach to information 
design and development, and the importance of information systems in providing 
managers with useful and valid data for policy development, program development, 
and program monitoring. The section is divided into two sub-sections, including (1) 
information system constraints, and (2) information system design. 

A. Information System Constraints 

In the 1991 state survey (Kiernan et al., 1991) that prompted the development of 
this Manual, 50% of the state MR/DD agencies responding reported perceived 
constraints on their information systems. These constraints included resources, lack of 
coordination across agencies or levels within the system, needed expansion/updating 
of the current system, confidentiality, quality of the data, and lack of a mandate 
regarding data use. Additionally, the overwhelming quantity of desired data was an 
additional constraint as reflected in comments such as, "we can't collect all the 
information that people would like", and "we have more data than information." 

These findings are consistent with what one finds in the information system 
literature regarding information systems that frequently are not understandable by 
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users, sometimes relevant to the decisions at hand, and frequently concentrating more 
on machine than people considerations (Brackett, 1987). Additionally, the experience 
of management information system users (e.g., Jacobson, 1991; Jacobson & 
Scheerenberger, 1985; Rouse, 1986; Vasta, 1985; Wasserman, 1980) suggests 
additional constraints including: 

‘ineffective communication between the user community and the 
developer; 

‘lack of knowledgeable end users; 

‘over optimism about development time, development cost, and/or 
operational costs of the system; 

‘lack of intermediate steps during which management and customers 
can review progress and determine the system's status; 

‘lack of user access, technical assistance, and output provision to end 
users; 

‘hardware selection and purchases based on budgetary criteria rather 
than data base volume, transaction volumes, or statistical manipulation; 

‘inability to perform inter-data base synthesis; 

‘inadequate training on form completion and data entry. Insufficient 
technical support. 

‘no systematic mechanism for translating policy questions into questions 
that can be dealt with by data bases; 

Overcoming these information system constraints is not easy. However, a 
number of suggestions for doing so are listed in Table 9. The 10 constraints listed in 
the table, along with the suggested solutions, underscore the importance of system 
design factors such as clear goals, understandable processes, and desired products. 
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Table 9 

Suggestions for Overcoming Common information System Constraints 
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These factors are elaborated upon in the following sub-section on information system 
design. 

B. Information System Design 

Information and information systems are advancing very quickly and, in many 
cases, faster than the user's ability to integrate the technology with the desired 
product. But knowledge in and of itself is insufficient; it must be organized in a way 
that permits answers to important questions and provides useful information to 
decision makers. In that sense, a state's information system should have clearly 
defined inputs, throughputs and outputs. Such an information system design model is 
presented in Figure 7. The model's five components provide the basis for discussing 
the significant factors that users of this Manual should consider as they attempt to 
improve their state's information systems related to day and employment programs. 

1. Information system goals. The previous section of the Manual discussed 
how data can be used for policy development, program development, and program 
monitoring. These uses should be the goals of an information system and thus drive 
the system's development, implementation, training, and use. 

Once the goals for the system are identified clearly, then subsequent steps can 
be taken. However, the importance of this first step cannot be emphasized too 
strongly, for unless one begins with the end clearly in mind, it is easy to end up in the 
wrong place or with data that do not provide managers with useful information. 

2. Information system Input . Once the system's goals are defined clearly, 
then the necessary input can be provided. However, before selecting the necessary 
data sets, consideration should be given to the information needs at different levels 
within the service delivery system: specifically, the service provider, the state, and the 
federal government. From this perspective, system designers need to ask seriously 
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what are the data needs at each of these three levels. A suggested answer is 
summarized below (Schalock & Hill, 1986): 

Service Provider (agency, county, region): Needs client referenced 
data including persons served, services provided, costs, and 
outcomes. 

State: Needs the ability to aggregate these critical data sets across 
service providers and service options and to evaluate the impact 
of the services provided. 

Federal: Needs aggregated data on service outcomes by consumer 
characteristics and results from (or potential for) impact analyses. 

This recommendation regarding shared roles in management information is 
based on the assumption that providers are responsible for client-referenced 
information; states are responsible for systems-level information and state-specific 
policies; and the federal government is responsible for state-level aggregation, state- 
level comparisons, and national policy development and evaluation. 

A dictum in information science is that, "the best information comes from the 
lowest level in the system." It was this concept that provided the basis for the provider- 
referenced data collection formats discussed in Part I (Figures 1-5). Thus, state-level 
information system staff should work closely with the local service provider to ensure 
that data collection formats and processes are clearly understood and used reliably. If 
this is done, it will result in greater confidence for inputting the following information 
data sets: 
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Policy Development 
Funding Patterns 
Consumer 

Characteristics 

Costs 

Outcomes 



Program Development 
Persons Served 
Services Provided 

Costs 
Outcomes 
Waiting Lists 



Program Monitoring 
Persons Served 
Services Provided 

Costs 

Outcomes 



3. Information system development . There are numerous definitions and 
models of information systems that are beyond the scope of this User's Manual. For 
the present purposes, there are four important facts that users need to keep in mind 
(and ask about) regarding their state's information system. These include 
(Wasserman, 1980): 

‘Information systems are used by persons who are unfamiliar with the 
technical aspects of computer hardware and software so that the 
underlying operating system, language processor(s), and hardware 
should be made as invisible and unobtrusive as possible. 

‘The way in which information systems are used and integrated into an 
organization's environment is at least as important as the way that they 
are constructed. 

‘Information systems involve access to and modification of large volumes 
of data, and thus must be organized with the "end in mind." 

‘The way in which humans interact with the system is of great importance 
and affects the selection and placement of terminals, hard versus soft 
copies, and turn around time. 
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These four facts, when interpreted broadly, underscore the need for clear design and 
development standards for a state's information system. Five such standards are 
summarized in Table 10. 

The actual development of the information system is very technical and requires 
knowledge of computer language, data bases, and database environments. Users of 
this Manual will probably not be "computer specialists"; however, one should 
understand the questions to ask, the data that are available, and the verification of 
standards related to reliability, simplicity, user friendliness, economy and evolvability. 
Entering into this dialog, plus involvement in the validation process described below, 
will help significantly in overcoming some of the knowledge and training-based 
constraints summarized in Table 9. 

4. Information system validation. Manual users need to consider two 
levels of validation: the software system and the data input system. 

A. Software system. In reference to the first type of validation, probably 50% of 
the development effort on a software system takes place after the software program 
has been written. Thus, the system must undergo a number of "tests" before it is fully 
operational. Three of these tests include (Brackett, 1987; Wasserman, 1980): (1) unit 
testing, in which individual program modules are tested for correctness and whether 
they meet their specification; (2) integration testing, in which two or more modules are 
joined together and tested together; and (3) acceptance testing, in which the user 
determines whether the system conforms to its specifications and goals. 

B. Data input system. Reference was made previously to a set of data 
selection and collection criteria that can be used to validate the data sets selected. 
These criteria and their definitions are presented in Table 11. In reference to these 
criteria, Manual users will want to ask such questions as, "Do the data reflect important 
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Table 11 



Data Selection and Collection Criteria And Their Definition 
Data Selection Criteria 

Attributed to program: an obvious link between services and 
outcomes. 

Sensitive to change and intervention: services affect potential 
outcomes. 

Obtainable: those measures that one has time and tools to actually 
measure. 

Objective: measures that can be easily quantified and thereby 
measured. 

Prioritized: measures that reflect the program's goals and objectives. 

Data Collection Criteria 

Accessibility: available from the primary source (e.g., service provider, 
funding agency). 

Completeness: availability of data element for all service recipients. 

Accuracy: reflective of actual service or outcome. 

Timeliness: whether data covers the period(s) of interest and how soon 
after such period(s) the data can be collected. 

Flexibility: degree to which the data element/set will be influenced by 
shifts in budgets, policies, research needs and/or quality of data 
source. 

Cost: time and resources required to collect and analyze the data 
element/set. 



attributes of the program or services provided?"; "Are the data objective?"; and "Are 
the data timely?" 

Additionally, user's need to validate the accuracy of the data submitted. The 
usual procedure for accomplishing this is to do a 10% (or more, if necessary) 
validation sampling in which the accuracy of the data submitted is checked against the 
primary data source. Corrections or adjustments are then made in the respective data 
sets. 

5. Data analysis . Manual users are encouraged to think broadly about data 
analysis, rather than simply about numbers, tables, and graphs. In this broader view, 
data analysis involves different levels including verifying, reporting, aggregating, and 
analyzing. Additionally, the broader view stresses the relationship among these 
levels, information system goals, and shared management information. 

A. Verifying . This level of analysis may be the most important because of its 
focus on error rate, missing data, the reliability and validity of the information system’s 
data, and the actual use of the data. If poor quality or low user rates are found, then 
problems in the system’s development or validation need to be addressed (see Figure 
7 for proposed feedback loops). 

B. Reporting. Reporting is a straightforward function that involves summarizing 
for the agency (and across agencies) data on the four critical data sets (persons 
served, services provided, costs, and outcomes). The typical statistics used for this 
function include mean, median, and standard deviation. 

C. Aggregating . A higher level of data analysis includes using a cross-tabs 
(matrix) approach to summarizing the critical data sets by some common factor such 
as personal characteristic (age, I.Q., gender), funding source (Title XIX vs. State 
General Fund), geographical location (urban vs. rural), or program type (supported vs. 
sheltered employment). The data collection formats presented in Part I provide the 
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data entry formats for these cross-tab analyses. 

D. Evaluating . This level of data analysis involves determining the significance 
or benefits from habilitation programs. The specific types of analyses conducted 
include outcome, impact, and benefit-cost (Schalock & Thornton, 1988). As discussed 
in Part II, this level of analysis is central to program and policy development and 
requires considerable resources related to comparison groups and complex statistical 
techniques. Frequently, this level of analysis is conducted by evaluators (for example, 
Federal contractors or university affiliated researchers) that are external to the state's 
information system. 

In summary, Manual users are encouraged to view the interactive relationship 
among these four levels of data analysis and the information system's goals and 
shared management perspective discussed previously. A model that shows this 
interrelationship is shown in Figure 8. 

Manual users should now be able to evaluate relevant aspects of their state's 
information system related to day and employment programs. Thus, the Manual 
concludes with the "Information System Assessment" which can be found in Table 12. 
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Table 12 



Information System Assessment 

Directions : Answer each of the following questions regarding your state's 
information system for day and employment programs. Questions asked are based on 
Part III of the User's Manual - 

Question Evaluation 

(check) 

Yes No Not Sure 



1 . Are there clear information system goals 
related to: 

a. Policy Development 

b. Program Development 

c. Program Monitoring 

2. Is there a clear focus related to: 

a. Shared Management Perspective 
(Provider, State, Federal) 

b. Data Sets (Persons Served, 

Services Provided, Costs, Outcomes) 

3. Are there design and development standards 
related to: 

a. Reliability 

b. Simplicity 

c. User Friendliness 

d. Economy 

e. Evolvability 

4. Are there validation procedures involving: 

a. Data Selection 

b. Data Collection 

5. Does the system have data analysis 
capabilities related to: 

a. Verifying 

b. Reporting 

c. Aggregating 

d. Evaluating 
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Footnote 1 



Although the term "cost" will be used throughout the Manual , it is important to 
point out that "costs" are not the same as "expenditures" or "rates", although these 
terms are often used as synonyms. Under prospective payment, standardized service, 
supervision, support, capital, maintenance and supply costs are combined through a 
rationale to compute rates, and then providers are reimbursed for anticipated 
expenditures through rates. Ideally, expenditures will mirror reimbursement by rates, 
but total expenditures will typically hover around the total of individual rates in cost 
centers with appropriate management control. It is important that a state's MIS is able 
to project future costs as a foundation for rate calculation (Jacobsen, personal 
correspondence). 
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